Summary

• build on PR #1060's GPTQ XSA11 bigram-hash architecture
• retune the schedule with WARMDOWN_ITERS=4000 and add a single-pass online token/within-word/word-start agreement evaluator packaged inside the record folder to improve bpb at eval time

What best_agree Does

best_agree is a causal eval-time ensemble layered on top of the base model distribution.

It maintains three prefix-only experts:

• token n-gram top-token hints
• within-word continuation hints
• word-start first-token hints

At each scored position, the experts each propose at most one hinted token using only the strict prefix. The system then picks the best hinted token and applies a boost to that token inside the model's normalized distribution. When multiple experts agree on the same token, it adds a small extra agreement boost. So the gain comes from agreement between causal experts, not from looking up the gold token or rescoring with future information.

Results

val_bpb: 1.11085863 (4-seed mean, std 0.00030217) | 15,953,221 bytes worst case | 8xH100 SXM

This improves the current README leader 1.1194 by 0.00592043 nats/byte and 0.00854137 bpb across four seeded runs.

A one-sided t-test confirms the improvement exceeds 0.005 nats/byte over 1.1194 with p = 0.00155 (t = 8.7892, df = 3), meaning there is only a 0.16% probability the observed gain is due to random chance.

Why This Online Cache Is Valid

Earlier cache-style evals often failed because they either:

• queried the cache using the realized next token or let x_t influence whether a cache hit existed
• blended only the realized token probability instead of defining a full normalized vocabulary distribution
• treated bucket-local counts or hash-bucket scores as if they were already normalized token probabilities

This implementation is different:

• it uses only the strict prefix to choose at most one hinted token h_t plus a prefix-derived confidence before x_t is consulted
• it starts from the base model's full softmax distribution and applies a one-token boost with renormalization, i.e. p'_t(a) = exp(beta_t * 1[a = h_t]) p_t(a) / Z_t
• it scores position t before updating the online state with x_t
• it evaluates in a single left-to-right pass over the full validation stream, in order

So this is a causal, normalized online overlay on top of the base model rather than a target-conditioned or unnormalized cache score.

Runtime

• 4-seed mean online eval wallclock: 467.78s (std 9.06s)
• the current implementation of the n-gram experts slows inference significantly, but this looks like an implementation issue rather than a fundamental limit and could likely be optimized a lot further

Test plan

• re-parsed the bundled seed logs and verified the README table fields against the logged metrics and total bytes
• recomputed BPB-to-nats conversions, sample mean/stddev, 95% confidence interval, and the one-sided t-test directly from the four bundled logs
• confirmed the bundled logs report online eval wallclock under the 10-minute budget on 8xH100

Across the different submissions and reruns I tried, these n-gram cache experts seem relatively consistent and typically give about a 0.003-0.004 bpb boost.